NFKD implementation #32
Conversation
| end | ||
|
|
||
| def compatibility_decomposition?(mapping) | ||
| COMPATIBILITY_FORMATTING_TAG_REGEXP =~ mapping.first |
There was a problem hiding this comment.
Unless I misunderstood the purpose of this method, it might be better for it to explicitly return true/false instead of an int/nil.
There was a problem hiding this comment.
I agree that it's a good practice to return true/false from ?-methods. This one is for internal use so I was less careful about this rule. Give me a couple of minutes, I'll fix it.
|
This generally looks absolutely awesome. I just commented on a few lines, but it looks nearly ready to merge in. Let's release a new version of the gem after merging as well. |
|
Hey, @camertron, I turned |
|
Hey @KL-7 yes, looks great. I'll merge it in. I'd like you to add those extra comments at some point, but that's not blocking the merge. |
Including NFKD implementation.
I added NFKD implementation. It ended up being the base class of normalizers hierarchy. I don't like it a bit, as I initially wanted to create some abstract base class for them, but things worked out this way. NFD works absolutely the same way, but skips compatibility decomposition. So it's implemented as a subclass of NFKD with a single overridden method.
During the process I splitted the algorithm into small methods that should make it easier to add NF(K)C algorithms later. Possibly we'll have to revisit the classes hierarchy, but in general it should be quite straightforward (assuming we have composition algorithm implemented) as these forms are build on top of NF(K)D algorithms.
Normalization specs are updated to test both algorithms. I didn't add specific tests for
NFKD#normalize. It'd be nice to have some simple test cases for it, but I need some valid normalization examples for that. I remember Andrew generated some of them using JRuby and Java Unicode library. Maybe I'll do the same thing a bit later.Besides that, I moved code points conversion methods into a separate utility module as in my opinion they are not a responsibility of the normalization classes.